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Abstract 


In 2015, 26 two- and four-year institutions received three-year grants to 
implement reforms consistent with the Integrated Planning and Advising for Student 
Success (iPASS) initiative. Grantees committed to launch or enhance existing 
technologies and undertake related organizational changes that would enable them to 
provide more effective advising and support to all students. CCRC was engaged as a 
research partner to document the iPASS colleges’ implementation efforts and outcomes. 

CCRC analyzed key performance indicators (KPIs)—including student retention 
from first to second semester and first-term grade point average—using administrative 
data collected from all 26 participating institutions. This report describes trends in KPIs, 
aggregated by college sector and cohort, in the participating colleges from 2011 to 
2017—from the period before the initiative started until about two years after the grants 
were awarded. Yet because iPASS reforms cannot easily be isolated from other reforms 
the colleges were carrying out, it is difficult to directly associate the modest changes over 
time in KPIs that we observe with iPASS efforts. This report also includes college scores 
on an iPASS development index, a weighted measure we created to gain insights into the 
level of adoption of iPASS technologies and practices across the 26 grantee institutions 
as of fall 2017, when colleges provided answers to an institutional survey about their 
efforts. We find that while no institution had fully implemented iPASS, at most 


institutions, substantial progress was being made. 
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CCRC’s Role in Three iPASS Research Projects 


The Integrated Planning and Advising for Student Success (iPASS) initiative—which has provided up 
to $225,000 to each of 26 colleges to help them adopt technologies for improving education planning, 
advising, and student risk targeting and intervention by 2018—was launched in 2015 with funding 
from the Bill & Melinda Gates Foundation and The Helmsley Charitable Trust. It followed on the 
heels of a similar initiative, undertaken from 2012 to 2015 at 19 colleges, in which several lessons 
were learned: 


e Emerging technologies have the potential to allow students to create and follow academic plans 
effectively, receiving support when they struggle. 

e Technology alone is not enough to achieve project goals. Deep changes in institutional 
structures, systems, and attitudes are required. 

e High-quality advising and student support may be facilitated through a set of core SSIPP 
principles, which call for advising to be sustained, strategic, integrated, proactive, and 
personalized. 


CCRC has been involved in both initiatives. Under the more recent initiative, EDUCAUSE and 
Achieving the Dream (ATD) have provided implementation services in the form of technical 
assistance to iPASS grantee colleges, while CCRC has conducted research on college activities and 
the student experience. All three organizations—EDUCAUSE, ATD, and CCRC—have sought to 
learn whether the reform of advising and student supports—made possible through the use of 
technology—provides students with a more seamless and holistic advising experience and ultimately 
improves student outcomes. 


As an evaluator and thought partner in the 2015-2018 iPASS initiative, CCRC has been engaged in 
three related research projects, which have resulted in reports, presentations, blogs, tools, and other 
resources for the field. 


Project 1. Measuring trends in development and scaling: CCRC has analyzed progress in 
implementation and student outcomes during the grant period across all 26 participating colleges. 
Resulting reports include a survey of technology use and advising practices provided to the colleges, a 
baseline report of key performance indicators (KPIs) (Armijo & Velasco, 2018), and a final report of 
trends in the KPIs after two years of project implementation (current report). 


Project 2. Understanding implementation: CCRC has studied implementation processes at nine 
colleges, some of which emphasized advising in STEM pathways. We conducted a review of the 
literature (Fletcher, Grant, Ramos, & Karp, 2016), reported on the use of predictive analytics 
(Klempin, Grant, & Ramos, 2018), released a set of case studies of four iPASS colleges (Klempin, 
Pellegrino, Lopez, Barnett, & Lawton, 2019), and studied how iPASS reform has unfolded at different 
levels of the college ecosystem (Klempin & Pellegrino, forthcoming). We also wrote an invited 
chapter on the SSIPP principles in practice (Klempin, Kalamkarian, Pellegrino, & Barnett, 2019). 


Project 3. Evaluating enhanced advising at three colleges: In collaboration with MDRC, CCRC 
has conducted research at three colleges that were provided technical assistance as they developed 
enhanced iPASS advising systems targeted to specific student populations. We partnered in an 
evaluation that included a randomized controlled trial and qualitative fieldwork to understand 
implementation at each college. This resulted in a report on the project designs developed at each 
college (Kalamkarian, Boynton, & Lopez, 2018), an interim report on early outcomes (Mayer et al., 
2019), a report on implementation (underway), and a final report on outcomes (planned). 


1. Introduction 


Improving completion rates in broad- and open-access colleges and universities 
has become a national imperative. In recent decades, data collected and published by the 
National Center for Education Statistics have indicated that these institutions graduate, on 
average, fewer than half of the students who enter them (e.g., McFarland et al., 2018). As 
these data became widely known, community colleges in particular began to undertake 
major efforts to increase their graduation rates (Cooper, 2017). 

Reform efforts aimed at improving institutional performance have addressed a 
range of issues, including low rates of completion of developmental coursework, 
inadequate monitoring of student progress, and insufficient student support services. But 
when researchers have taken stock of innovations and initiatives meant to address these 
issues, they have often found that piloted reforms proved difficult to scale or resulted in 
little or no change in outcomes (Zachry Rutschow et al., 2011; Quint, Jaggars, Byndloss, 
& Magazinnik, 2013). Further, resources to enact or scale promising interventions 
remained scarce in many U.S. states. Thus, the experiences of most enrolled students 
were not improving. 

One reform approach that has emerged in response to the limitations of some of 
the previous efforts is technology-mediated advising. This strategy aims to improve 
student learning, persistence, and credential completion by transforming the advising and 
student support experience at a relatively modest cost. In 2015, the Bill & Melinda Gates 
Foundation and The Leona M. and Harry B. Helmsley Charitable Trust provided three- 
year grants to 26 two- and four-year institutions to implement Integrated Planning and 
Advising for Student Success (iPASS). Grantees committed to launch or enhance existing 
technologies and undertake the related organizational changes that would enable them to 
provide more effective advising and support to all students. Under the iPASS initiative, 
EDUCAUSE and Achieving the Dream (ATD) provided the colleges with technical 
assistance to support implementation. 

The Community College Research Center (CCRC) was engaged as a research 
partner to document and describe the iPASS colleges’ implementation efforts and 


outcomes (see box on previous page). To study implementation, a series of site visits and 


interviews were conducted, and an institutional survey was administered. To assess 
student outcomes,! CCRC analyzed key performance indicators (KPIs)—including 
student retention from first to second semester and first-term grade point average—using 
administrative data collected from all 26 participating institutions. Analysis of the KPI 
data is the primary subject of this report. 

An earlier report on iPASS KPIs presented baseline descriptive student outcomes 
data for the colleges from 2011 to 2014, the period just prior to the grant awards (Armijo 
& Velasco, 2018). The indicators, which will be discussed in greater detail below, 
measure aspects of student momentum and progress to completion that could potentially 
be improved through advising reform. The first report showed that there was wide 
variation in individual institutions’ baseline indicators, with the largest differences 
occurring between two- and four-year institutions; it also showed that between 2011 and 
2014, the measured KPIs showed little to no significant change. 

This report serves as an extension of the earlier KPI report by describing trends in 
the participating colleges from 2011 to 2017. First, we provide background on the 
rationale for technology-mediated advising and the tools that are typically adopted by 
colleges engaged in such reform. We also outline a theory of action that shows how 
elements of technology-mediated advising reform may produce improvements in student 
experiences and outcomes. Next, we describe trends in selected KPIs aggregated by 
college sector and cohort. We then introduce the iPASS Development Index, a weighted 
measure we created to gain insights into the level of adoption of iPASS technologies and 
practices across the 26 grantee institutions as of fall 2017, when colleges provided 
answers to an institutional survey about their efforts. We conclude by briefly discussing 


the findings. 


'Tn collaboration with MDRC, CCRC is also conducting a randomized controlled trial to evaluate the 
impact of an enhanced form of technology-mediated advising on students; three of the 26 participating 
colleges are participating in that study (Mayer et al., 2019). 


2. iPASS Design 


2.1 The Rationale for iPASS 

College students at broad-access institutions, and community college students in 
particular, often face multiple academic and nonacademic barriers to success, including 
inadequate educational preparation, the need to manage family responsibilities while 
enrolled, and the similar need to balance work and school commitments (Porter & 
Umback, 2019). In addition, these students, many of whom are the first in their families 
to attend college, may lack college know-how, may not have clear education or career 
goals, or may not understand how college can help them achieve their goals (Karp, 2011). 
More access to advising support could help students address some of these barriers. 

There has long been recognition that guidance and support for students at broad- 
access colleges is insufficient (Grubb, 2006). Colleges generally offer various sources of 
support, but students are typically expected to explore them on their own. Further, 
advising and orientation are often not made mandatory for students, and the understaffing 
of advisors at institutions is a very common phenomenon (Bailey, Jaggars, & Jenkins, 
2015). Such features preclude extended or frequent in-person meetings between advisors 
and students, and advising sessions are often focused on course registration for first-time 
students. 

The research literature offers guidance on what effective advising under ideal 
circumstances should look like. Several scholars recommend a “developmental” approach 
to advising that takes place over a sustained period of time. Under this model—also 
referred to as advising-as-teaching—the advisor does not focus solely on course 
registration but instead seeks to develop the advisee’s reasoning, problem-solving, and 
analytical skills as the student engages in program exploration and goal-setting (Karp, 
Kalamkarian, Klempin, & Fletcher, 2016; Crookston, 1972). Additional studies suggest 
there are benefits to a “SSIPP” approach, which emphasizes advising that is sustained, 
strategic, integrated, proactive, and personalized (Karp & Stacey, 2013; Kalamkarian, 
Karp, & Ganga, 2017). 

But how are advising-as-teaching and the SSIPP approach possible in colleges 


that are short on advisors and resources? The aim of iPASS is to allow advising and 


student support staff to more closely follow these models by gaining efficiencies through 
the use of technology tools. Ideally, using technology enables college personnel to 
communicate with more students on a regular basis and better connect them to the 
information and supports they need to progress to graduation (Karp et al., 2016). 

There is a wide variety of technology tools available; in 2019, the student 
supports technologies and services market was worth $560 million and included more 
than 200 companies (Bryant, Seaman, Java, & Chiaro, 2019). There are various ways to 
describe these tools (Karp et al., 2016; Bryant et al., 2019), but here we group them into 
four categories: (1) risk targeting and intervention or “early alert” systems, which help 
identify students who are struggling in order to promptly intervene; (2) education 
planning systems, which include tools for selecting programs and courses, mapping 
degree plans, and tracking progress toward degree completion; (3) communication tools 
used to reach out to students with needed information in a timely way; and (4) predictive 
analytics tools, which use student data to provide information to advisers on students that 
might need particular types of help (Klempin, Grant, & Ramos, 2018). Ideally, the 
products integrate several functions. 

While the existing evidence on iPASS is mixed, some early research suggests that 
components of iPASS reforms can positively affect student outcomes, particularly when 
the technology-mediated intervention incorporates an element of personal interaction or 
support (Kalamkarian, Karp, & Ganga, 2017). Yet, research also shows that substantial 
organizational commitment is necessary to implement iPASS reforms as they are 
designed. Simply deploying a new technology in an institution does not guarantee that the 
student experience will be changed for the better (Mayer et al., 2019). The technology 
must be integrated into the daily work of the college staff to provide an opportunity for 
substantive improvement in student experience and outcomes. This requires both broad 
organizational change and change in practice among individuals (Karp & Fletcher, 2014; 


Karp et al., 2016). 


2.2 The iPASS Theory of Action 
Figure | shows the iPASS theory of action (adapted from Mayer et al., 2019). 


Ideally, when technology tools and high-quality advising practices are implemented as 


intended, advisors are better able to provide student support at scale and in an 


individualized way. As advisors work with students, students gain knowledge and a 


greater ability to make decisions and adopt new behaviors. For example, they set career 


and academic goals, develop an academic plan that aligns with those goals, and learn and 


use time management and study skills that facilitate the accomplishment of those goals. 


This should lead to improved student outcomes, measured by KPIs, such as greater 


retention rates, more credits earned, higher grades, and higher graduation rates. 


Resources 


Trained student 
support staff 


Institutional leaders 
who provide staff 
members with 
technology tools and 
professional 
development 


Technology tools, 

such as: 

° early alert systems 

* predictive 
analytics 

* education 
planning tools 

* communication 
tools 


iPASS Theory of Action 


Program Activities 


Advisors regularly 
communicate with 
students, in-person 
and using technology 
tools 


Early in the semester, 
advisors and faculty 
use technology tools 
to identify struggling 
students 


Advisors intervene 
with identified 
students, virtually and 
during required 
advising appointments 


Advisors use 
technology tools to 
refer students to 
support services 


Advisors use technology 
tools to teach students 
to think critically about 
academic and career 
goals 


Advisors use technology 
tools to document and 
share notes from 
advising sessions 


Figure 1 


Mediators 


Clear academic and 
career goals 


Understanding of how 


current courses align 
with long-term goals 
Support from peers, 
faculty, and staff 
members 


Improved time-man- 
agement and study 
skills 


Enrollment in required 
courses for program of 
study in optimal 
sequence 


More attempted credits 
that align with program 


of study 


Outcomes 


Short-term Long-term 


Students earn 
more credits 


Students complete 
— a certificate or 
degree 


Students continue 
to enroll 


Students have a 


Students earn F A 
i higher cumulative 
higher grades GPA 


Note. Figure adapted from iPASS Logic Model in Mayer et al. (2019, p. 8, Figure 2.1). 


3. KPI Data and Student Characteristics 


Our original intent in this study was to collect administrative data from the 
colleges to examine whether the funded iPASS reforms were associated with 
improvements in KPIs. However, our ability to link changes in KPIs to particular reforms 
is hindered by several factors. First, implementation of technology-mediated advising 
reform can take a long time. Colleges must not only launch technologies, but they must 
also integrate those technologies into the daily work of administrators and staff. Colleges 
also need to refine or reform their advising practices so that student support follows the 
SSIPP and advising-as-teaching models. These sorts of major shifts in practice require 
fundamental organizational and behavioral change. Yet our KPI data ends after only two 
years of participation by the colleges in the initiative. 

In addition, findings from an online survey administered to the colleges in this 
study in the summer of 2017 reveal that many institutions were involved in reform 
initiatives to improve student outcomes other than iPASS during the iPASS grant period. 
Twenty institutions, for example, were carrying out developmental education reforms, 
and 18 were undertaking “guided pathways,” a college-wide reform effort for increased 
student success that, among other things, aims to bring changes to student advising. In 
addition, a number of colleges had already begun implementing advising reforms prior to 
the start of the iPASS grant period. These conditions do not allow us to attribute changes 
in KPIs among the participating institutions to iPASS activities alone. Thus, the KPI data 
presented here must be taken as descriptive of overall trends that may be influenced by a 
variety of factors. 

We focus on descriptive KPI trends for the 26 iPASS colleges between fall 2012 
and fall 2017, which corresponds to the period before and during the first two years of the 
funded initiative. The colleges received their three-year iPASS grants some time during 
2015; we consider the beginning of the grant period as fall 2015. The colleges worked 
with a third-party organization to create the datasets used in our analyses, which were 
stored on a secure website and transmitted to CCRC after quality checks. Institutions 
uploaded the data over five cycles between the spring of 2016 and the spring of 2018. 
These data were supplemented with data from the Integrated Postsecondary Education 


Data System (IPEDS). 


To ensure comparability with the preliminary report associated with this project 
(Armijo & Velasco, 2018), we followed the same procedures as those used in the first 
report to define the sample and variables. Hence, we limit our sample to first-time-in- 
college and non-dually enrolled students who entered college between fall 2012 and fall 
2017. Data are aggregated by cohort, with all students entering in one academic year 
(e.g., in fall 2012 and spring 2013) counted together as one cohort (e.g., the 2012 cohort). 
However, the 2017 cohort includes only students who entered in the fall, so this cohort 
does not appear in figures that pertain to full-year outcomes. We also aggregate the 
student-level data by institution and create cohort average metrics for each college for the 
years before and during the iPASS grant. 

To provide context, Table 1 displays descriptive statistics by college sector for 
students who entered college a year before the start of the iPASS initiative in the 2014 
cohort. The students at the two- and four-year institutions differ in several important 
ways. For example, the two-year institutions have a much greater percentage of part-time 
students than the four-year institutions (76% vs. 20%) and a larger percentage of students 
from within the state (96% vs. 80%). The two-year institutions also serve a smaller 
percentage of students who are 19 years old or under at the time of enrollment (68% vs. 
91%), a greater percentage of first-generation students (37% vs. 28%), and a greater 
percentage of Black (18% vs. 11%) and Hispanic (17% vs. 10%) students. These kinds of 


differences between two-year and four-year institutions are fairly typical nationally. 


Table 1 


Characteristics of First-year Students Enrolled in iPASS Colleges in 2014, by Sector 


Variable 


Two-year college (%) 


Four-year college (%) 


Enrollment intensity in first term 


Full-time (12 or more credits) 24 80 

Part-time (fewer than 12 credits) 76 20 
Residency 

In-state 96 80 

Out-of-state 2 15 
Age 

19 and under 68 91 

20-24 16 4 

25 and older 16 5 
Gender 

Female 51 53 

Male 47 47 
Race/ethnicity 

American Indian 0 

Asian 10 

Black 18 11 

Hispanic 17 10 

Mixed race/ethnicity 13 

Native Hawaiian 0 

Non-resident alien 8 

White 32 44 

Missing 20 3 
First-generation 

First-generation student 37 28 

Not first-generation student 28 61 

Missing 36 12 
Neighborhood income (based on student census track) 

Household income less than $35,000 30 12 
Number of first-year students in 2014 32,957 51,850 
Number of institutions 12 14 


4. Descriptive KPI Trends by Sector 


Here we describe KPI trends for the 26 iPASS colleges, classified according to 
the outcomes included in the theory of action. We first present the following short-term 
student outcomes: first-year credit accrual, first to second year retention rate, and first- 
year GPA. We then present these long-term outcomes: cumulative GPA and degree 
completion rate. For each outcome, we display figures aggregated by sector and cohort. 
In other words, we look at the outcomes of interest for the cohort of students that enrolled 
in either two- or four-year institutions in each of the years shown in the horizontal axis of 
the accompanying figures. In terms of identifying trends that are statistically significant, 
we use a t-test of difference in means with clustered standard errors at the institution 
level. We refer to a difference in means as statistically significant when its associated p- 
value is less than 0.1, which denotes that the difference has only up to a 10% likelihood 
of being observed by chance. Appendix Table Al shows differences in means and 


standard errors for all KPIs examined. 


4.1 Student Short-term Outcomes 
Credit accrual 

Here we examine three indicators of credit accrual: credit momentum (attempted 
15 credits in the first term), average number of credits earned during the first academic 
year among students who first enrolled in the fall, and percentage of credits attempted 
that were earned during that year. 

Figure 2 presents trends on students’ credit momentum. Students at four-year 
institutions were much more likely to attempt 15 credits or more during their first term 
than students in two-year institutions. When looking at the trends for two-year 
institutions, however, we observe a pattern of increasing credits earned over time. To 
examine the extent to which the increase is significant, we conduct a hypothesis test to 
evaluate whether the results observed during the last year of available data are 
statistically different from the results in 2014, the year prior to the receipt of the iPASS 
grant. We find that while 7% of the students who enrolled in two-year institutions in fall 
2014 attained credit momentum, 14% of two-year students in the cohort entering in fall 


2017 did so. Unlike at four-year institutions where the difference in this KPI between the 


two cohorts (57% vs. 54%) is not significant, the difference in credit momentum at the 


two-year institutions between the 2014 and 2017 cohorts is statistically significant. 


Figure 2 
Percentage of Students With Credit Momentum 
Two-year Colleges Four-year Colleges 
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Note: Credit momentum is defined as attempting 15 or more credits during the first term. 


Figures 3 and 4 present credit accrual patterns during the first academic year. 
Overall, students at two-year institutions earned fewer credits, in total and as a percentage 
of the credits attempted, than students at four-year institutions. Nevertheless, as with 
credit momentum, there is a pattern of increasing credit-earning over time at the two-year 
colleges that is not observed at four-year colleges. Specifically, the average number of 
credits earned in the first year for entering cohorts at two-year institutions increased by 3 
credits over the 2012—2016 period. The percentage of credits attempted that were earned 
in the first year at two-year colleges also increased, from 56% to 61%. However, neither 


of these differences is statistically significant. 
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Figure 3 
Average Number of Credits Earned in Year 1 
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Figure 4 


Percentage of Credits Attempted That Were Earned in Year 1 
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Retention 

Figure 5 depicts the percentage of students who continued to be enrolled in their 
second year of college for each cohort. (We limit this analysis to cohorts up to and 
including 2015 in order to observe retention through the entire 2016 academic year.) 
Overall, retention rates were higher at four-year institutions than at two-year institutions, 
and with little change over time. During the period of interest, we find that two-year 
institutions retained between 48% and 49% of students. The difference between the 2013 
and 2015 retention rates is not statistically significant at two-year institutions. At four- 
year institutions, the percentage of students that continued to be enrolled from their first 
to their second year increased slightly over the same period, from 78% to 79%; the 


difference is statistically significant. 


Figure 5 
Percentage of Students Retained in Their Second Year 
Two-year Colleges Four-year Colleges 
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First-term GPA 

Figure 6 presents each cohort’s first-term GPA. In the time period examined, we 
observe a small upward trend in the average first-term GPA at both types of institutions. 
At two-year institutions, the average first-term GPA increased from 2.03 in 2014 to 2.12 
in 2017, but this increase is not statistically significant. At four-year colleges, the average 


student GPA increased from 2.82 in 2014 to 2.95 in 2017, which is statistically 


significant. 
Figure 6 
Average First-term GPA 
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4.2 Student Long-term Outcomes 

The long-term outcomes we consider are cumulative GPA at two- and four-year 
colleges and associate degree completion rates at two-year colleges for each cohort. We 
exclude bachelor’s degree completion rates as our data do not span enough years to 
measure this. As we do for the short-term outcomes, we conduct hypothesis tests 
comparing whether the differences in the results for pre- and post-iPASS cohorts are 


Statistically significant, clustering standard errors at the institution level. 


Cumulative GPA 
Figure 7 shows that the average cumulative GPA was higher for students at four- 


year institutions than for students at two-year institutions. At two-year institutions, 


13 


cohorts had an average cumulative GPA of approximately 2.20 throughout the period of 
interest, which does not change significantly over time. At four-year institutions, the 
average student cumulative GPA climbed from 2.85 in 2014 to 2.94 in 2016, an increase 


that is statistically significant. 


Figure 7 
Average Cumulative GPA 
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Degree completion 

The final outcome we examine is the degree completion rate at two-year 
institutions. Figure 8 presents the percentage of students from each cohort who completed 
an associate degree within two (left graph) or three (right graph) years. We focus on two- 
year colleges and exclude the 2015 and later cohorts from the analysis because those 
students had not yet had three full years to earn an associate degree. This means that all 
the students that we examine at two-year institutions began college before the start of the 
iPASS grants (1.e., before fall 2015). 

We find a decreasing trend in the percentage of students completing an associate 
degree in two years (6% in 2011 vs. 4% in 2014). The percentage of students completing 
a degree in three years is relatively more stable over time at about 10%. The differences 
in completion between the 2011 and the 2014 cohorts are not statistically significant for 


either the two- or three-year to completion outcomes. As discussed in Armijo and 


Velasco (2018), these results are lower than the average degree completion rates at 


community colleges nationally. 
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5. Implementation of iPASS Reforms 


The successful implementation of technology-mediated advising requires 
significant changes in practice by institutions and their personnel. Colleges must invest 
time and resources in launching technologies and integrating them into the work of 
faculty and staff, making them a part of institutional culture. To better understand 
implementation progress under the iPASS initiative, we create an index to roughly 
measure the extent to which technology-mediated advising practices had been developed, 
deployed, and integrated within each institution, using data from the 2017 institutional 
survey. The index provides useful context for interpreting the KPI trends described in the 


previous section. 


5.1 Using the Institutional Survey Responses 
We compute the index scores using data gathered from an online survey 
conducted at the institution level in the summer of 2017, approximately 18 months after 


most institutions received their iPASS grants. It is important to recognize that institutions 
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completed the survey when they were still relatively early in the process of implementing 
this complex initiative. Each survey was completed by a college staff member, typically a 
provost, vice provost, dean, or director of student services. The survey had a 100% 
response rate.” 

The survey was designed to document institutions’ advising structure, practices, 


and reform efforts in eight categories: 
1. Advising structure: the institution’s overall advising model. 


2. Advising practices: the presence of specific practices such 
as making referrals to support services and the use of case 
management, and the length of time any of these practices 
had been in place. 


3. Professional development: trainings offered as a part of the 
iPASS initiative. 
4. Use of data: the use of predictive analytics and other data to 


improve advising. 


5. iPASS goals and accomplishments: the college’s iPASS 
goals, the extent of college staff participation in the project, 
and which technologies had been adopted and when. 


6. Technology scale of adoption: the extent to which the 
college used iPASS technologies. 


7. Outcomes: the outcomes the institution expected iPASS to 
affect. 


8. Institutional context: other ongoing reform efforts or 
student success initiatives. 

In creating and computing the index (see below as well as Appendix B), we use 
question responses from the survey that focus on the adoption of four technologies—(1) 
early alert systems, (2) predictive analytics, (3) education planning tools, and (4) 
communication tools—and on 13 specific high-quality advising practices—(1) using case 


notes, (2) making referrals to support services, (3) proactive outreach to students to set up 


2 A document describing the results of the survey is available upon request. 
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advising appointments, (4) differentiated support based on student need, (5) using a case 
management approach, (6) multi-semester course planning, (7) linking course planning to 
transfer and/or career planning, (8) following up with students identified as at-risk, (9) 
using learning outcomes for advising, (10) using automated messaging to target groups of 
students, (11) using predictive analytics and reporting of student and institutional data to 
improve advising, (12) requiring students to see an advisor after the first semester, and 
(13) other high-quality practices. We also use question responses that show the extent to 
which the four technologies have been integrated into advising practice. 

While the institutional survey offers a rich source of data on iPASS 
implementation, limits on the number and types of questions included mean that some 
important aspects of iPASS were not explored. In addition, we are unable to observe 
whether student behaviors depicted in the theory of action were actually adopted. 
Furthermore, the survey, and thus the index, represents a snapshot in time and does not 
capture all changes induced by the iPASS grants. Thus, the index may be viewed as 
shedding light on important iPASS practices and technologies that were developed and in 
place at the colleges at a particular time, but not as a complete assessment of the level and 


quality of implementation at participating colleges. 


5.2 Creating the iPASS Development Index 

The CCRC research team first created a draft index based on our knowledge of 
the initiative’s goals and priorities. To gain feedback on our efforts, we convened two 
webinar panels of student support professionals and researchers, presented to them the 
draft index, its relationship to particular survey questions, and our methods of 
computation. We asked them to provide input on the dimensions and sub-dimensions of 
the index, the relative importance of each and thus how heavily particular aspects of 
reform should be weighted and computed. We synthesized the feedback we received and 
incorporated changes, which resulted in the index as described here. We explain how we 
use the survey responses and how we compute the index in greater detail in Appendix B. 

The index has three dimensions, each of which is weighted differently, based on 


these attributes (which themselves involve weighting): 
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e Weighted number of high-quality advising practices 
implemented, i.e., advising practices associated with the 
SSIPP and advising-as-teaching models (13 practices 
possible); 


e Weighted number of technologies used, which captures the 
number of technology types implemented at the institutions 
(4 types possible); and 


e Extent of integration of these technologies into advising 
practice, which aims to identify the level to which advising 
practices incorporated the use of the 4 types of 
technologies. 

Figure 9 depicts the basic score structure of the index. We attribute 50% of the 
total score to the number of high-quality advising practices implemented, while the 
number of technologies used accounts for 15% of the total score, and the integration of 
technology into advising is weighted as 35% of the total. Survey responses from each 
participating college were weighted accordingly, allowing us to calculate the score for 
each dimension as well as the overall score. The result is a profile that provides insights 
into the level of development of iPASS practices and technologies as of the summer of 


2017. 
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Figure 9 
iPASS Development Index Score Structure 


Dimension 3 Score 

Based on extent of integration of the four technologies into 
advising practice 

(Integration of communication tools, early alerts, and 
education planning each constitute 30% of the dimension 3 
score; integration of predictive analytics constitutes 10% of 
the dimension 3 score.) 

Dimension 3 score of 0-1 constitutes 35% of the total index 
score. 


Dimension 2 Score 

Based on weighted number of four technologies used 

(Using communication tools and using early alerts are both 
weighted at 1. Using education planning technology is 
weighted at 1.5. Using predictive analytics is weighted at 0.5.) 


Dimension 1 Score Dimension 2 score of 0-1 constitutes 15% of total index score. 


Based on weighted number of high-quality advising practices 
implemented 

(Linking course planning to transfer and/or career planning is 
weighted 2.5 times that of other practices.) 

Dimension 1 score of 0-1 constitutes 50% of total index score. 


5.3 Index Findings 

Figure 10 shows the rank of institutions according to their index score, separating 
two-year and four-year institutions. In this ranking, an institution reporting that it had 
enacted and integrated all practices and technologies across the index dimensions would 
earn a score equal to one. Conversely, a score close to zero for an institution would show 
that few iPASS-related practices or technologies were in place. The blue-shaded bar 
displays the overall score on the index, and the three dots display the scores for each 


dimension. We include a table version of this figure as Appendix Table A2. 
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Figure 10 
iPASS Development Index and Dimension Scores by Institution 
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The index scores indicate that no institution had fully implemented technology- 
mediated advising reforms at the time the survey was administered. However, there are 
no institutions with a score below 0.3. This suggests that all the funded iPASS institutions 
had practices in place that were indicative of some level of implementation. Overall, we 
found that most of the institutions were in middle or upper stages of development: 20 out 
of the 26 institutions had a score above 0.5 on the index. We also observed a wider 


variation in development across four-year than across two-year institutions. Specifically, 
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the institutions with the highest and the lowest index scores are both four-year 
institutions, while two-year institutions are concentrated in the 0.43 to 0.83 score range. 

The scores associated with the three index dimensions reveal some interesting 
patterns. For example, most of the institutions had adopted several technologies, and 
some had adopted all four of them. Nevertheless, some institutions that have a high score 
on “number of technologies” have mid- or low-ranking overall scores because they had 
not integrated these technologies into advising at a high level or had few high-quality 
advising practices. Moreover, the results indicate that even top-ranking institutions had 
not fully integrated technologies into advising. Specifically, only one institution had a 
value above 0.75 on the dimension concerning integration of technologies into advising, 
while four institutions had scores less than 0.25 on this dimension. 

The iPASS Development Index findings align with prior literature describing the 
challenges involved in carrying out technology-mediated advising reform (Kalamkarian, 
Karp, & Ganga, 2017). Even with the availability of grant resources for launching 
advising technologies, these reforms are complex and take time to implement. The 
findings suggest that approximately 18 months after institutions received their iPASS 
grants, they were at varying stages of implementing advising reforms at scale. These 
results are also consistent with the preliminary findings described in CCRC’s initial KPI 
report (Armijo & Velasco, 2018), which indicated that, prior to the initiative, there was 


wide variation in terms of technology adoption. 


6. Discussion and Conclusion 


This report considers activities undertaken by 26 two- and four-year institutions 
that received grants in 2015 to participate in iPASS, an initiative to launch or enhance 
existing technologies and undergo related organizational changes that would ideally 
enable the colleges to provide more effective advising and support to all students. 

To better understand changes in student outcomes over the period before and 
during the grant, we aggregate KPIs by cohort and college sector and examine them over 


multiple academic years (between four and six years, depending on the KPI), typically 
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identifying outcomes for the 2012 through 2016 student cohorts. There is a modest 
positive trend over time in credit momentum, in credits earned in the first year, and in 
percentage of attempted credits earned in the first year among students at two-year 
institutions. However, the differences over time are statistically significant only for credit 
momentum at two-year institutions. There is also a modest positive trend for first-year 
retention and first-term GPA of students in two-year and four-year institutions, but the 
differences are statistically significant only at the four-year institutions. And in terms of 
two- and three-year associate degree completion rates at two-year institutions, we find an 
overall decreasing trend, though the differences over time are not statistically significant. 

We use the iPASS Development Index to provide insights on the colleges’ 
adoption of technologies and practices across this complex advising initiative. The index 
helps us identify variation in implementation levels across the 26 colleges. It suggests 
that after approximately 18 months of the iPASS grants, all participating institutions 
exhibited a number of practices associated with technology-mediated advising. While no 
institution had fully implemented iPASS, at most institutions, progress was being made. 

The design of this study does not permit an analysis of causal relationships, so we 
do not know the extent to which any of the modest changes in KPIs we find are due to 
iPASS reform efforts. Nor do we know whether changes in KPIs are associated with any 
of the particular practices or technologies that have been introduced. Many of the 
colleges were simultaneously pursuing other major reform efforts, which further 
complicates interpretation of the findings. Our results provide only a descriptive picture 
of the progress of students in these colleges over the time period under study. 
Nevertheless, as findings from our index show, more time is likely needed for the iPASS 
reforms to take hold and to scale them to all students. 

We recommend that future research and development of technology-mediated 
advising focus on testing and refining particular components of the reform approach. 
Among other topics, research should examine the different mixes of technology tools that 
institutions are adopting and the forms and amounts of staff training that is offered to use 
them well. What is more, research should attempt to identify those technologies that have 
potential to lead to substantial changes in practice. Research should also address whether 


and how specific advising practices and activities bring about the changes in student 
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behaviors that are hypothesized to lead to improved student outcomes. Finally, 
researchers should document how technology-mediated advising is being incorporated 
into guided pathways and other reform initiatives underway in many community colleges. 


Such studies could go a long way to inform and direct a field that is in the midst of 


widespread transformation. 
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Appendix A 


Means and Standard Errors for KPIs 


Table A1 
Differences in Means in the Cohort KPls Prior to and During iPASS Participation 


Mean differences 


Cohorts Two-year Four-year 
Variable compared institutions institutions 
Short-term outcomes 
Credit momentum 2014 vs. 2017 0.054* 0.012 
(0.026) (0.075) 
Number of credits earned 2014 vs. 2016 0.808 1.177 
(1.054) (1.149) 
Percentage of attempted credits earned 2014 vs. 2016 -0.011 0.039 
(0.015) (0.027) 
Percentage of students retained 2013 vs. 2015 0.003 0.017* 
(0.010) (0.009) 
First-term GPA 2014 vs. 2017 0.004 0.238* 
(0.065) (0.111) 
Long-term outcomes 
Cumulative GPA 2014 vs. 2016 -0.042 0.088*** 
(0.046) (0.021) 
Percentage of students earning associate 2011 vs. 2014 -0.015 NA 
degree in two years (0.013) 
Percentage of students earning associate 2011 vs. 2014 -0.002 NA 
degree in three years (0.013) 


Note. The last two columns present differences in mean KPls before and during iPASS implementation, by college 
sector. Standard errors are in parentheses and clustered at the institution level. 


# D <01, ** p<.05,* p<. 


27 


Appendix B 


iPASS Development Index 


Here we explain how we use the responses to the institutional survey to compute 


the development index dimension and total scores. 


1. Institutional survey questions used for each of the index dimensions 

The iPASS practices index measures three dimensions of technology-mediated 
advising implementation based on: (1) the weighted number of high-quality advising 
practices implemented, (2) the weighted number of technologies used, and (3) the level of 
integration of these technologies into advising practice. In this section, we describe how 
institutional survey questions and answer choices are used to compute dimension scores. 


Detailed descriptive statistics for the survey questions are available upon request. 


1.1. Number of high-quality advising practices 
The institutional survey asked about high-quality advising practices in place or in 
the process of being implemented at the institution. For the following list of practices, 


survey respondents selected all that applied: 
a. Using case notes 
b. Referrals to support services 


c. Proactive outreach to students to set up advising 
appointments 


d. Differentiated support based on student need 


e. Case management (i.e., meeting with the same student over 
time) 


f. Multi-semester course planning 


g. Linking course planning to transfer and/or career planning 
[counts as 2.5 times the weight of the other practices] 


h. “Closing the loop” (follow-up with faculty and staff who 
identify student as at risk) 
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i. Using learning outcomes for advising 
j. Automated messaging to targeted groups of students 


k. Using predictive analytics and reporting of student and 
institutional data to improve academic advising 


1. Requiring students to see an advisor at any point after first 
semester 


m. Other 


We take the weighted number of high-quality advising practices as the central 
indicator for the extent to which high-quality advising practices have taken place at the 
institutions; thus, this dimension accounts for 50% of the total possible index score. This 
appraisal was affirmed by the panel of experts we consulted, who also highlighted the 
importance of giving a higher weight to the practice related to career planning (practice g 
above), which we count as 2.5 times more important than each of the remaining practices. 

To compute the responses and transform them into a dimension score for advising 
practices, we give a value of | to each of the practices selected (except for practice g, 
which we give a value of 2.5), add them, and divide the sum by the maximum possible 


value. This process can be summarized by the following equation: 


Q 
Scale = Xq=1%a 


() 


where qg indexes the practice, and x, takes a value of 1 (or 2.5 in the case of practice g) if 
the practice was selected and 0 if it was not. 


For example, if a given institution checked nine out of the 13 advising practices 
listed in the survey, and those practices included linking course planning to transfer 


and/or career planning, then the score for the advising practices dimension would be: 


14+41+1414+0+04+25+1+1+0+0+1+1 #105 
# of Adv. Pract. Scaled = ee ay re ee ee were 


1.2. Number of technologies 
This dimension counts the number of technology types an institution had adopted 


by the time of the survey. Specifically, the question from the institutional survey reads: 
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Which iPASS technologies has the college implemented (either homegrown or 
through an outside vendor), at any point in time—not just for the current grant? 


Predictive analytics 
Communication platform 
Early alerts 

Degree audit 

Education planning 


i a Sd 


All institutions reported using degree audit technology. Because its use is 
ubiquitous and because its exclusion does not affect the ranking of institutions, we 
eliminate this technology type from further computation and focus on the four remaining 
technologies. Prior evidence on predictive analytics indicates that though many 
institutions have implemented it, there are concerns with respect to the validity, 
interpretation, and ethics of its use (Klempin, Grant, & Ramos, 2018). These points were 
raised by the panel of experts, who also highlighted the importance of education planning 
tools in student advising. Hence, our weighting of the responses gives higher importance 
to using education planning technology than to using predictive analytics. 

Similar to what we do for the advising practices dimension mentioned above, we 
attribute a positive value when the institution reported having a technology type and 0 
when it did not, add the responses together, and divide by the maximum value possible. 
We incorporate the relative importance of some technology types for advising over others 
by attributing them different weights. Specifically, if the institution reported using 
predictive analytics, we give that response a value of 0.5. If the institution reported using 
education planning technology, we give that response a value of 1.5. We give responses 
for the other two technologies (communication platform and early alerts) a value of 1.0 


each. The value in the denominator is thus 4. 


1.3. Integration of each technology into advising 

To assess the extent to which each technology implemented has been integrated 
into advising, we consider each technology type mentioned just above as a sub-dimension 
in the third dimension of the index. We use questions from the institutional survey to 


compute separate values for early alerts, predictive analytics, education planning, and 
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communication tools usage. Then, we weight and add these to obtain a score for the 


dimension on integration of technologies into advising. 


1.3.1. Early alerts 
To capture the extent to which early alert systems have been adopted by the 


institutions we use two questions from the institutional survey: 


How does your institution intervene with students who are flagged? — Select all 


that apply 
a The early alert system sends an automated message to the student. 
b. Advisors are assigned cases for follow-up.? 


Cy Faculty receive communication indicating that flagged students received 
an intervention. 


If the institution marked 3 out of the 3 options available, we consider it as having 
a high level of intervention. If it marked 2 out of 3 we consider it as having a medium 
level. If it marked 1 out of 3, we consider it as having a low level of intervention. 


The second question we use considers the use of early alert tools by faculty: 


Percentage of faculty raising flags each semester: 


a. 0-10% 

b. 10-25% 
c. 25-50% 
d. 50-75% 
é 75-100% 


In the case of early alerts, we give more relevance to how the institution 
intervenes when students are flagged than to the proportion of faculty raising flags each 
semester. While the first question provides evidence about the extent to which the 
institution has implemented procedures for closing the loop with flagged students, the 


second question may also involve the extent of student needs at each institution. For 


3 Here we combine two response options: (1) Advisors are assigned cases for follow-up, but institutions do 
not follow a systematic process for how advisors intervene, and (2) Advisors are assigned cases for follow- 
up and there is a systematic process in place at the institution for how advisors intervene. 
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question I, we give values of 3, 2, and 1 for high, medium, and low levels of 
interventions, respectively, and divide the result by 3. For question II, we give a value of 
5 when the reported percentage of faculty raising flags each semester was between 75% 
and 100%, 4 when it was between 50% and 75%, 3 when it was between 25% to 50%, 2 
when it was between 10% to 25%, and 1 when it was between 0% and 10%. Then we 
divide by 5 to preserve the number in a 0 to | range. To account for the relative 
importance of question I over question II, we multiply the question I value by 1.5 and the 


question II value by 0.5, add the results together, and divide by 2. 


1.3.2. Predictive analytics 
In this case, we use two questions from the survey about the breadth of use and 


purpose of predictive analytics tools available at the institutions. 


I. Which stakeholders at your institution regularly utilize predictive analytic tools? 
(select all that apply) 
a. Advising director(s) 
b. Institutional research staff 
res Advisors 
d. Senior administrators (1.e., vice provosts, provost, vice presidents, 
president) 
G Deans and/or associate deans 
ft Other student support staff 
g. Faculty 
h. Other staff 
II. Institution uses analytics and reporting of student and institutional data to 


improve academic advising (Y/N). 


With regard to question I, we recognize that the use of this technology type by 
some stakeholders has a higher potential impact for improving student advising than its 
use by others. Thus, we attribute a higher weight to the use of predictive analytics tools 
by advising directors and advisors than by institutional research staff or other staff. 

To compute values for the responses to question I, we again use equation | (above). 
We give a weight of 1.5 to the use of the technology by advising staff and a weight of 0.5 


to the use by research and other non-listed staff. Responses to question II have a value of 1 
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when yes and 0 otherwise. To obtain the final value of each institution for this sub- 


dimension, we add the computed values for questions I and II and divided by 2. 


1.3.3. Education planning 

In this case, we focus on the use of education planning tools by students. 
Specifically, we look at two questions: the proportion of students that have planned at 
least one semester of coursework using the institutional degree planning tool and the 
percentage of students that have planned all semesters. As the latter question informs 
about both use and consistency of use, we attribute a higher importance to it for the final 
computation of this sub-dimension value. We supplement this data with institutions’ 


responses on the use of degree planning tools for transferring credits to other institutions. 


I. What proportion of students have planned at least one semester of coursework 
using your institutional degree planning tool? 


a. No degree planning tool 
b. 0-10% 
G: 10-25% 
d. 25-50% 
Gs 50-75% 
fi, 75-100% 
II. What proportion of students have planned all semesters toward completion of 


their degree in your institution using a degree planning tool? 


a. No degree planning tool 
b. 0-10% 
(eA 10-25% 
d. 25-50% 
C. 50-75% 
1 75-100% 
IH. ‘Institution uses the degree planning tool to allow students to check which courses 


to transfer to a specific institution (Y/N). 


For questions I and II we proceed in the same way as we do in the case of early 
alerts. That is, we give a value of 5 when the percentage was between 75 and 100 percent, 
4 when it was between 50 and 75 percent, 3 when it was 25 to 50 percent, 2 when it was 


between 10 to 25 percent, and 1 when it was between 0 and 10 percent. Then we divide 
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by 5 to preserve the number in a 0 to | range. In the case of question II, we give a value 
of 1 to a positive response and 0 otherwise. To account for the relative importance of 
question II over I, we multiply the value obtained for question I by 0.5 and the value of 
question II by 1.5. To obtain the final value of each institution for this sub-dimension, we 


add the computed values for questions I, II, and II and divide by 3. 


1.3.4. Communication tools 
We use two yes/no questions as the measure of the extent to which 


communication tools have been integrated to advising practices: 


I. Advisors utilize your institution ’s iPASS technologies to take notes on advising 
Sessions. 
Il. The note-taking process is a consistent practice among advisors. 


Our goal is to capture if communication technologies were being used to take 
notes in advising sessions and the consistency of that practice across advisors in the 
institution. Similar to what we do with the yes/no questions elsewhere, we attribute a 
value of 1 to yes responses and 0 otherwise, add the values of questions I and II, and 
divide by 2. 

Once we compute each of the sub-dimension index values, we weight them and 
add them together to obtain the score for the dimension on integrating each technology 
into advising. Specifically, we multiply the values obtained for early alerts, education 
planning, and communication tools by 0.3; we multiply the value of predictive analytics 
by 0.1. This is consistent with the computation of the dimension on technologies 
implemented, where we acknowledge the relative smaller importance of using predictive 


analytics. 


2. Computation of the iPASS Development Index Scores 
Table B1 displays the descriptive statistics of the dimension scores and sub- 
dimension values that constitute the iPASS development index score. It is based on the 


responses to the survey of the 26 colleges participating in this study. 
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Table B1 
Descriptive Statistics of the iPASS Development Index Dimensions and Sub-Dimensions 


Standard 

Mean deviation Min Max 

No. of high-quality advising practices 0.761 0.156 0.345 1.000 
No. of technologies implemented 0.750 0.258 0.100 1.000 
Integration of each technology into advising 0.465 0.193 0.075 0.801 
Early alerts 0.452 0.343 0.000 1.000 
Predictive analytics 0.537 0.230 0.000 0.906 
Education planning 0.245 0.285 0.000 1.000 
Communication tools 0.673 0.373 0.000 1.000 


The final score was obtained by applying the formula 


(1) iPASS development index, = (0.5 * advising practices,) + (0.15 * 
tech. implemented,) + (0.35 * integration of techs to advising,) 


where, 


(2) Integration of techs to advising index, = (0.3 * early alerts,;) + (0.1 * 
pred. analytics, ) + (0.3 * ed. planning,) + (0.3 * comm. tools,) 

In equation 2 we acknowledge the importance that advising practices overall have 
for improving student support by giving that dimension half of the weight of the index. 
We also incorporate the idea that technologies that are not integrated and do not 
complement advising do little to improve student support by giving the technologies 


implemented dimension a small weight in the overall index. 
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Table B2 
Development Index Score by Institution 


Development index score 


Sector College (0 to 1 scale) Dimension scores (0 to 1 scale) 
Number of Integration of each 
advising Number of technology into 
practices technologies advising 
J 0.83 0.86 1 0.72 
H 0.8 0.79 1 0.71 
S 0.78 0.93 0.6 0.65 
R 0.76 0.86 0.9 0.55 
O 0.75 0.79 1 0.58 
L 0.71 0.86 0.7 0.49 
ee c 0.68 0.79 1 0.38 
D 0.67 0.86 0.8 0.34 
AA 0.55 0.73 0.7 0.24 
U 0.52 0.73 0.5 0.24 
K 0.48 0.66 0.2 0.35 
F 0.47 0.55 0.7 0.27 
N 0.43 0.34 0.8 0.38 
G 0.93 1 1 0.8 
P 0.86 0.93 1 0.69 
T 0.81 0.82 1 0.72 
I 0.76 0.86 0.8 0.61 
V 0.73 0.93 0.5 0.53 
A 0.72 0.68 1 0.64 
Rea a M 0.71 0.93 0.5 0.48 
B 0.68 0.73 1 0.47 
Y 0.62 0.79 0.8 0.28 
X 0.57 0.68 0.8 0.31 
Ww 0.49 0.52 0.7 0.37 
Q 0.43 0.59 0.4 0.21 
Z 0.32 0.55 0.1 0.08 
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